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Teaching mathematics has long required the use of technol¬ 
ogy due many powerful affordances. More recently, education 
technology has been developed to support personalized learn¬ 
ing through the use of adaptive learning systems. Through the 
use of educational technology in online learning, there is great 
potential for improving students’ mathematics achievement. 
In this article, we report the results of an evaluation study, 
where 11 online mathematics educational technology products 
were distributed to close to 200,000 K-12 students and their 
teachers in the state of Utah to supplement classroom instruc¬ 
tion. While only ten percent of students used the products at 
the recommended level over the course of the 2014-15 school 
year, there were six products where an educationally mean¬ 
ingful impact on mathematics achievement was found. While 
teachers responded positively, a third of teachers reported 
lack of access to technology as a barrier. We are already see¬ 
ing improved usage during the second year of the project due 
to modifications to the expectations for schools based on what 
was learned from the first year of implementation. 
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INTRODUCTION 

Education technology shows great potential for facilitating mathemat¬ 
ics instruction and improving students’ mathematics learning outcomes 
(Cheung & Slavin, 2013). In this article, we report the results of an evalua¬ 
tion study, where 11 mathematics educational technology products were dis¬ 
tributed to close to 200,000 K-12 students and their teachers in the state of 
Utah. These products were to be used as supplements to regular classroom 
instruction and were not intended to replace the instruction by teachers. In 
particular, we report students’ usage of these products over the course of the 
2014-15 school year, discuss the findings of the impact of these mathemat¬ 
ics products on students’ achievement on the state assessment, and describe 
teachers’ feedback about using these products. We also discuss the scien¬ 
tific and practical significance of this project and how it might inform future 
research and similar projects across the nation where technology is being 
scaled across a large group of teachers and their students. 

BACKGROUND 

Teaching mathematics has long required the use of technology due to 
the educational benefits afforded by such technology (National Council of 
Teachers of Mathematics., 2011). In recent years, educational technology 
has provided many features that enable it to support individualized instruc¬ 
tion and personalized learning (e.g., adaptive learning system). Specifically, 
these kinds of products provide each student with different learning content 
(e.g., adaptive quiz and real-time feedback) and curriculum by collecting and 
analyzing each student’s data so that each student can learn mathematics at 
their own pace. These technologies are being used as effective supplements 
for teachers, allowing them to monitor each student’s progress and current 
state of mathematics performance. To realize these advantages and to im¬ 
prove their students’ mathematics achievement, many school districts have 
been increasing their spending on educational technology (Richard & Sara- 
beth, 2016). 

The more that students use educational technology to learn, and the more 
that schools invest in such technology products, the more precisely the as¬ 
sessment of the effects of such technology is needed. To date, a substantial 
number of studies have attempted to examine the effectiveness of educational 
technology on student achievement in mathematics (see Cheung & Slavin, 
2013; Slavin, Lake, & Groff, 2009). Most recently, De Witte, Haelermans, 
and Rogge (2015) found a positive effect of computer-assisted instructional 
programs on learning outcomes in mathematics for secondary students. How¬ 
ever, Cheung and Slavin (2013) argue that the effects could vary depending 
on technology type and usage intensity. Therefore, more research is constant¬ 
ly needed to assess the effects of technology products rigorously. 
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Preparing and supporting teachers to integrate technology effectively is 
also important in maximizing the effect of implementing new technology 
(Lawless & Pellegrino, 2007; Niess, 2005; Thomas & Palmer, 2014). In¬ 
deed, the roles and knowledge of teachers in technology-rich classrooms 
are much different than they used to be in the traditional classroom (Trigue- 
ros, Lozano, & Sandoval, 2014). For example, when students learn using a 
mathematics software program, they might need their teacher’s assistance 
in order to properly interpret the feedback they receive from the program 
(Abboud-Blanchard, 2014). For adequate training and sufficient support for 
mathematics teachers, it is essential to identify what factors exist that make 
it difficult or easy for teachers to use technology to teach mathematics. 

Prior research has discovered different types of factors that may influ¬ 
ence the successful integration of technology. Hew and Brush (2007) have 
identified the following six main barriers to technology integration for K-12 
classrooms from forty-eight empirical studies: lack of resources (40%), lack 
of knowledge and skills (23%), the institution (14%), teacher beliefs and at¬ 
titudes (13%), student assessment (5%), and subject culture (2%). Lack of 
resources, a barrier mentioned most frequently in this study, includes sev¬ 
eral sub-categories, such as lack of technology, limited access to technol¬ 
ogy, lack of time, and limited technical support (Hew & Brush, 2007). Even 
though the lack of sufficient resources has long been a main obstacle to the 
effective integration of technology into teaching and has been considered a 
barrier that is easier to eliminate (Ertmer, 1999), it continues to be a signifi¬ 
cant hindrance today (Carver, 2016). 

Previous research has also demonstrated that teachers’ attitudes toward 
and beliefs about the educational benefits of technology comprise another 
key factor in achieving the more frequent use of technology in classrooms 
(Domingo & Gargante, 2016; Inan & Lowther, 2010; Kim, Kim, Lee, Spec- 
tor, & DeMeester, 2013; Petko, 2012; Yuen & Ma, 2008). The theoretical 
background of these studies is based on the Technology Acceptance Model 
(TAM), which suggests that a user’s willingness to use new technology in¬ 
creases when the user perceives that the new technology is easy to use and 
is useful (Davis, 1989). Through a qualitative survey of 68 K-12 teachers, 
Carver (2016) found that the majority of the responding teachers (82%) re¬ 
ported that the use of technology increased student engagement and student 
understanding. 

Prior studies, however, have explored mainly teacher perception about a 
wide variety of technology devices (e.g., computer, handheld device, vid¬ 
eo project, camera, etc.) used in their classrooms, rather than focusing on 
personalized, adaptive learning software. Each device could be utilized dif¬ 
ferently for teaching and learning depending on its own attributes and the 
characteristics of the subject matter. Moreover, the recent work of Howard, 
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Chan, and Caputi (2015) has confirmed that teachers’ beliefs about technol¬ 
ogy integration differs between subject areas (e.g., English, mathematics, 
and science). Therefore, more research is needed to identify how teach¬ 
ers incorporate digital learning software into their classrooms, specifical¬ 
ly when the software is designed for personalized learning and especially 
when such software is used in a particular subject area, such as mathemat¬ 
ics. 


INTERVENTION 

Through funding from House Bill (HB) 139 legislation in Utah, the 
STEM Action Center was put into place in the Governor’s Office of Eco¬ 
nomic Development. The mission of the STEM Action Center is to promote 
STEM best practices in education with a focus on Utah STEM industry con¬ 
nections. Through funding from HB 139 (2013) and HB 150 (2014), the 
Utah state legislature appropriated $13.5 million for mathematics software 
for K-12 students for the 2014-15 and 2015-16 academic years. In summer 
2014, the STEM Action Center released a request for proposals (RFP) for 
providers of K-12 mathematics technology programs that provide a system 
that is adaptive and personalized to meet individual student needs, provides 
real time reporting to teachers and students, provides supports to address 
student needs, and a minimal amount of professional development or train¬ 
ing focused on how to use the product and reporting features. In Table 1, we 
provide a list of the 11 products selected through the RFP process, the prod¬ 
uct provider/vendor, and the grade levels in which they were implemented. 

In general, the products selected were intended to cover the curriculum 
for a particular grade level, with some flexibility to allow students to ac¬ 
cess below and above grade level curriculum with the exception of three 
products: EdReady, Reflex, and ST Math. EdReady is a program that is pri¬ 
marily used to assess a student’s readiness for college level mathematics. 
Once the assessment is completed and areas of weakness are highlighted, 
the student can use curriculum available within the program or use other 
curriculum provided by their teacher to address these identified areas of 
need. Reflex is a program that is used to develop mathematics fact fluency, 
such as addition and multiplication facts. ST Math is a completely visual 
spatial program with no oral or written language. It does not cover some 
grade level curriculum standards, such as geometry, but focuses on foun¬ 
dational conceptual understanding of number (whole numbers, decimals, 
and fractions) and operations. It also has the breadth of exposing students to 
algebra. All of these products were intended to be used as supplements to 
the core mathematics curriculum. 
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Table 1 

Overview of the Mathematics Products and Their Providers 


Product name 

Provider name 

Grade Levels Implemented 

ALEKS® 

McGraw-Hill 

3-12 

Catchup Math 

Hotmath, Inc. 

6-12 

Cognitive Tutor® 

Carnegie Learning 

9-12 

EdReady® 

NROC Project 

9-12 

i-Ready® 

Curriculum Associates 

K-8 

MathXL® 

Pearson NCS 

9-12 

Odyssey Math® 

Compass Learning 

6-8 

Reflex® 

Explore Learning 

6-8 

ST Math® 

MIND Research 

K-8 

SuccessMaker® 

Pearson NCS 

K-5 

Think Through Math® 

Think Through Learning 

K-8 


Due to a delay in contract negotiations, SuccessMaker was not imple¬ 
mented during the 2014-15 academic year. While licenses for Odyssey 
Math were distributed, the teachers did not believe it was appropriate for 
their students and decided to not implement the product. 

PURPOSE AND RESEARCH QUESTIONS 

The purpose of this evaluation was to examine the impact of different 
mathematics technology products on students’ achievement and to under¬ 
stand teachers’ experiences implementing these products. This study ad¬ 
dresses the following questions: 

1. What is the impact of supplemental use of mathematics educational 
technology on student proficiency on the state assessment? 

2. What common themes do teachers report from the implementation of 
the educational technology in their classrooms? 
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METHODS 


Study Design 

The most rigorous research design for estimating the impact of a pro¬ 
gram would be to randomly assign either schools, teachers, or students to 
use the products or to continue business-as-usual (BAU) practices. Such 
studies allow one to make the strongest causal statements regarding the 
impacts of an intervention or program. However, in this case, random as¬ 
signment was not feasible, since the goal of this legislatively funded project 
was to reach as many students as possible and not to add on additional con¬ 
straints that might prevent some districts or schools from participating. 

Therefore, to answer the first research questions, the impact is estimat¬ 
ed through a quasi-experimental approach using propensity-score match¬ 
ing (PSM). In PSM, a comparison group is formed by matching individual 
students using the technology on a one-to-one basis to students who most- 
closely resemble these students in the BAU group. This is the next best ap¬ 
proach when random assignment is not feasible. 

To answer the second research question, we collected qualitative self- 
report data through surveys of teachers participating in the grant program. 
Classroom observations were not conducted due to the cost associated with 
collecting that type of information. 

Data Collection 

We collected monthly usage data from the providers to understand imple¬ 
mentation during the year. At the end of the year, we asked the providers to 
determine a benchmark for what was termed “fidelity of implementation,” a 
recommended amount of usage or level of mastery that they would expect to 
be associated with student achievement gains. In this way, we were able to 
see which students met this benchmark and determine if there was a greater 
impact for these students. Each product provider determined what represent¬ 
ed fidelity (recommended usage) for their product (see Table 2). 


Table 2 

Recommended Usage Benchmarks Set by Product Providers 


Product 


Grades 


Description of Benchmark 


ALEKS 

Cognitive Tutor 
EdReady 
Catchup Math 


6-8, 9-12 


K-5, 6-8, 9-12 
9-12 


9-12 


Minimum of 480 minutes (8 hours) 

An algorithm determined by the provider, not defined 
Not applicable * 

An algorithm determined by the provider, not defined 
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Table 2, Continued 


Product 

Grades 

Description of Benchmark 

i-Ready 

K-5, 6-8 

30 minutes per week 

Math XL 

9-12 

Not available 

Odyssey Math 

6-8 

Not available 

Reflex 

6-8 

An algorithm that includes fluency gains and average 
number of logins per week. 

ST Math 

K-5, 6-8 

An algorithm based on content progress and/or lab logins 
that differed by grade. 

SuccessMaker 

K-5 

Not available ** 

Think Through Math 

K-5, 6-8 

>20 lessons passed 


Note. * “Not applicable” is noted for EdReady, because the teacher determines appropriate usage; 
therefore, no benchmark for recommended usage was provided. ** “Not available” is noted when 
providers did not set a benchmark for recommended usage. 

To address the first question concerning the impact of these mathematics 
products, we received data from the Utah State Office of Education with the 
students in the grant program flagged and the rest of the students in the state 
to be used for comparison purposes. The de-identified data file we received 
contained the state achievement test (2014-15) scale score and proficiency 
level for mathematics, language arts, and science and other variables, such 
as unique student identifier, unique school identifier, school locale, student 
characteristics (gender, ethnicity. English Language Learner status, eligibility 
for free or reduced lunch, special education status), and prior achievement of 
each student on the state assessment (2013-14). 

To address the second research question concerning teachers’ feedback 
on these mathematics products, we developed a teacher survey to understand 
their satisfaction with the product, any concerns that they had with the prod¬ 
uct implementation, experience with the data reporting features, in addition 
to any other feedback they wanted to share. The survey included the follow¬ 
ing questions: 

1. Please describe how you used the technology product in the last thirty 
days (e.g., as a supplement, as an intervention, selected materials for instruc¬ 
tion, selected materials for homework). 

2. Describe your overall satisfaction with the technology product. 

3. Describe any technology or other barriers that prevented you from us¬ 
ing the product with your students as much as you would have wanted to use 
the product. 

4. Describe how you have been using any of the data reporting features 
of the product to understand your students’ progress and/or to inform your 
instruction. 
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Qualtrics, an online survey platform, was used to administer the survey and 
automatically record teachers’ responses. 

Analytic Sample 

By the end of the 2014-15 academic year, 196,625 licenses had been dis¬ 
tributed to K-12 students. However, there were only 152,276 with evidence 
of usage. In Table 3, we provide a summary of license distribution and us¬ 
age across the mathematics products. Differences in number of licenses 
distributed was based on the products requested in the district/school grant 
application. Across products, only nine percent of students met the recom¬ 
mended benchmark for usage as set by the provider (Table 3). 

However, there were data from only 44,497 students used in our analysis, 
who had both the prior and current year assessment data. One reason for the 
reduced sample is that students in K-3 did not have a prior year baseline 
assessment, and many students in grades 11 and 12 had completed their re¬ 
quired assessments. In addition, some parents choose to have their students 
opt out of taking the state assessment. These factors reduced the size of the 
sample in our analysis. 


Table 3 

License Distribution and Usage for the 2014-15 School Year 




Total Licenses Distributed 


Product 

Usage 

Percentage 

Percent 

Product 

Students 

Districts 

Charters 

Schools 

Usage in 
May 

Meeting 

Fidelity 

Benchmark 

ALEKS 

106,530 

26 

27 

299 

77,766 

73% 

2% 

Cognitive 

Tutor 

917 

3 

0 

3 

782 

85% 

10% 

Catchup 

Math 

286 

0 

3 

3 

82 

29% 

67% 

EdReady 

498 

4 

1 

7 

498 

100% 

NA 

i-Ready 

17,389 

12 

6 

74 

15,322 

88% 

4% 

MathXL 

3,124 

5 

3 

16 

3,085 

99% 

NA 

Reflex 

4,378 

5 

3 

20 

3,421 

78% 

44% 

ST Math 

36,327 

12 

5 

99 

31,162 

86% 

16% 

Think 

Through 

Math 

27,176 

8 

4 

94 

20,158 

74% 

32% 

Total 

196,625 

36 

38 

488 

152,276 

77% 

9% 


Note: Schools at times have used two products, therefore, the sum of the values for number of schools 
across products does not correspond to the total column value. NA=Not Available from the provider. 
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For three of the products there was insufficient student data to conduct an 
impact analysis: Cognitive Tutor, EdReady and Reflex. In Table 4, we report 
the sample sizes of the full sample and the fidelity sample if available for 
the remaining six products and the average usage information to address the 
first research question. The number of students in the fidelity sample is ap¬ 
proximately ten percent of the students in the full sample. 

Table 4 

Sample Size and Usage for Full and Fidelity Analytic Samples 



Full Analytics Sample 

Fidelity Analytic Sample 

Product 

Number of Students 

Average Usage 

Number of Students 

Average Usage 

ALEKS 

27,190 

835 minutes 

633 

2,329 minutes 

Catchup Math 

254 

86 minutes 

32 

474 minutes 

i-Ready 

3,981 

302 minutes 

190 

1,317 minutes 

MathXL 

318 

1,670 minutes 

- 

- 

ST Math 

5,858 

20 lab logins 

801 

76 lab logins 

Think Through 
Math 

6,896 

19 lessons 

2,814 

70 lessons 

Total 

44,497 

- 

4,470 

- 


The sample for the second research question included any teacher who 
completed a survey during the 2014-15 school year. We do not have infor¬ 
mation on how many total teachers participated in the grant program, since 
grants were awarded to districts and student licenses were distributed to 
schools. It was estimated that between 3,000 and 5,000 teachers were in¬ 
volved in the grant program. An overview of the number of teachers re¬ 
sponding to the survey by product is in Table 5. 

Table 5 

Sample of Teachers Completing Survey by Product and Overall 


Product 

Number of Teachers Completing Survey 

ALEKS 

1,216 

Cognitive Tutor 

15 

Catchup Math 

5 

EdReady 

12 

i-Ready 

462 
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Table 5, Continued 


Product 

Number of Teachers Completing Survey 

MathXL 

60 

Reflex 

97 

ST Math 

830 

Think Through Math 

236 

Total 

2,933 


Data Analysis 

We conducted the impact analysis based on two analytic samples: the full 
sample of students including students with any evidence of product usage 
and the fidelity sample of students including students who met the recom¬ 
mended usage benchmark. We matched students in the grant program using 
mathematics educational technology and those from the comparison group 
(BAU) using the Matchlt package in R through propensity score matching. 
Matchlt is used for nonparametric approaches to making causal inferences. 
This approach selects matched samples from the original intervention and 
BAU groups on propensity scores, the likelihood that the student would be 
in the intervention group. 

The data office of the state office of education recommended against 
combining scale score data across grades, due to differences in the construc¬ 
tion of each assessment. Instead they recommended that we match students 
based on prior year assessment scale scores in groups by type of test (e.g., 
mathematics grade 3), and then use proficiency as the outcome, which al¬ 
lowed the combination of data across grades. Following this guidance, we 
used the nearest-neighbor matching algorithm with 1-to-l matching for each 
test combination using the model: intervention ~ mathematics pretest scaled 
scores. We tested different approaches to matching (including some more 
complex models), but in the end we decided that our goal was to minimize 
pretest differences between the groups while using the same matching mod¬ 
el for each grant for ease of interpretation and dissemination of the results to 
our stakeholders. 

We merged the data sets across tests in order to analyze the impact of 
technology use on the proportion of students meeting proficiency, regardless 
of grade. We compared the students’ characteristics and prior achievement 
by product to determine if there were any significant differences between 
groups. A t-test was used to compare continuous variables and a z-test of 
proportions was used for all dichotomous variables. For most products, the 
groups were equivalent on prior assessment scores, but some differences 
were observed in demographic characteristics (as shown in Table 6) which 
were controlled for in our impact analysis. Flowever, for ease of reporting, 
due to our stakeholder audience, we used consistent models described next. 
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Table 6 

Statistically Significant Baseline Differences 


Product 

Full Sample 

Fidelity Sample 

ALEKS 

FRPL, SPED, WT, HSP 

GDR, WT, HSP 

Catchup Math 

LAP, FRPL, SPED 

- 

i-Ready 

FRPL, WT, HSP 

ELL, HSP 

MathXL 

LAP 

- 

ST Math 

LAP, FRPL, WT, HSP 

LAP, FRPL, WT, HSP 

Think Through Math 

FRPL, SPED, HSP 

FRPL, WT, HSP 


Note: FRPL = Free/Reduced Price Lunch, LAP = Language Arts Proficiency, SPED = Special Educa¬ 
tion, ELL = English Language Learner, GDR = Gender, HSP = Hispanic, WT = White. 

We conducted a logistic regression using proficiency on the mathemat¬ 
ics posttest as a binary outcome variable. The predictors in the logistic re¬ 
gression model were the following student-level variables: intervention, 
eligibility for free or reduced price lunch, special education status, English 
Language Learner status, gender, and prior year proficiency level for math¬ 
ematics and language arts state assessments. Proficiency levels were treated 
as categorical variables, where “1” was the lowest proficiency and “4” was 
the highest proficiency level. Standard errors were corrected for clustering 
of students within schools using bootstrapping with school ID as a stratum. 
The outcome, an odds ratio, indicates whether the impact is in favor of the 
group of students using the technology or the BAU group. The result ex¬ 
plains that the group in favor is more likely to have met proficiency on the 
state assessment. We also include a Cox effect size, which can be used to 
compare to effect sizes which have been shown to be educationally mean¬ 
ingful in prior research, such as 0.16 found by the meta-analysis by Cheung 
and Slavin (2013). 

For the second research question, we used open coding to develop codes 
and categorize teachers’ survey responses (Strauss & Corbin, 1998). We re¬ 
port themes that emerged from the analysis along with exemplar statements 
for each theme. We coded the same feedback from teachers for each prod¬ 
uct. The teacher feedback is important, because it provides important insight 
into their experiences and can inform the work in future years of the project. 
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RESULTS 


Impact Analysis 

There were 44,497 students included in the analysis for the full sample 
of students with any amount of usage and 4,470 students in the fidelity sam¬ 
ple of students who had met or exceeded the recommended level of usage. 
The results from logistic regression are provided in Table 7. We report odds 
ratio, standard error, p-\ alue, effect size, and 95% confidence interval of the 
odds ratio for each product. 


Table 7 

Impact of Technology use on Achievement by Product 


Product 
and Sample 

Sample 

Size 

Exp(B) 
odds ratio 

Standard 

error 3 

Significatnce 

level 

Effect 

Size 

95% Confidence 
Interval for Exp(B) 
odds ratio 

Lower Upper 

ALEKS 

Full Sample 

27,190 

1.014 

0.026 

0.607 

0.01 

0.964 

1.067 

Fidelity Sample 

633 

1.354 

0.144 

0.032 

0.18 

0.967 

1.897 

CatchUo Math 

Full Sample 6 

254 

1.294 

0.278 

0.333 

0.16 

0.730 

2.293 

i-Readv 

Full Sample 

3,981 

0.983 

0.063 

0.804 

-0.01 

.861 

1.122 

Fidelity Sample 

190 

2.765 

0.279 

0.002 

0.62 

1.410 

5.423 

MathXL 

Full Sample 6 

318 

1.464 

0.317 

0.078 

0.23 

0.821 

2.611 

ST Math 

Full Sample 

5,858 

1.125 

0.126 

0.296 

0.07 

0.910 

1.390 

Fidelity Sample 

801 

1.483 

0.435 

0.179 

0.24 

0.849 

2.590 

Think Throuah 

Math 

Full Sample 

6,896 

1.191 

0.177 

0.239 

0.11 

0.891 

1.593 

Fidelity Sample 

2,814 

1.339 

0.235 

0.097 

0.18 

0.952 

1.884 


aThe Standard Error and Significance have been adjusted for clustering. 

bThere were only 32 students in the fidelity sample for Catchup Math, which was too small of sample 
to include in the analysis. 

cThere was no fidelity sample for MathXL, because the product provider, Pearson, did not provide a 
fidelity benchmark. 












Effects of Educational Technology on Mathematics Achievement 


217 


When interpreting the odds ratio, values greater than 1.0 favor the stu¬ 
dents using the technology and anything less than 1.0 favors students in the 
comparison group. As can be seen from the odds ratio in Table 7, except for 
the i-Ready full sample, all other odds favor the students using the math¬ 
ematics products, which implies that these products play a positive role in 
improving students’ learning achievement or chances of meeting proficiency 
on the state assessment. The standard error and the related p-value of statis¬ 
tical significance are the values adjusted for clustering. There were only two 
products where the impact was statistically significant {p <.05), the fidelity 
samples for ALEKS and i-Ready, where students were using the products at 
the recommended level. 

A p-value is influenced by sample size and variance in the sample. A re¬ 
searcher could have a really large sample and get a significant yj-value, but 
the actual difference might not be very meaningful. Therefore, in research 
many are now using an effect size to better understand if there are mean¬ 
ingful differences. For mathematics educational technology, when used as 
a supplement to regular curriculum, prior research has shown the effect size 
to be around 0.16 (Cheung & Slavin, 2013). Therefore, we use 0.16 for the 
point of comparison to understand whether the impact of use of these prod¬ 
ucts is educationally meaningful. We list the effect size information in Ta¬ 
ble 7. All six products included in the impact study, had at least one sample 
reach or exceed the 0.16 effect size level for having an effect on the propor¬ 
tion of students meeting proficiency on the state mathematics assessment. 

Teacher Feedback 

To address the second research question, we discuss the findings from 
our analysis of survey data from 2,933 teachers in the grant program. Feed¬ 
back was provided for nine of the 11 products. We summarize the most 
frequent types of comments for each of the survey questions. The teachers 
were first asked to explain how they used the mathematics software prod¬ 
ucts for their teaching. Overall, 56% reported supplemental use and 28% 
reported use as an intervention (as shown in Table 8). 

There are some differences in use by product. For example, teachers us¬ 
ing MathXL reported primarily assigning it as homework (53%). Reflex is 
a product specifically used for mathematics fact fluency, which is why it 
makes sense that 30% of teachers’ reported the primary use as developing 
skill fluency. We did not gather more specifics on the ways in which teach¬ 
ers used the product as a supplement to instruction. In the second year of 
implementation, we are conducting site visits with interviews, focus groups, 
and classroom observations to better understand the supplemental use of 
these technology products to support mathematics instruction. 
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Table 8 

Common Ways Teachers Used the Technology (Percent of Teachers) 


Supplement Intervention or 


Categories 

to instruction 

Differentiation 

ALEKS 
(N=1216) 

49 

26 

Catchup Math 
(N=5) 

40 

40 

Cognitive 

Tutor (N=15) 

100 

13 

EdReady 
(N=12) 

17 

8 

i-Ready 

(N=462) 

47 

42 

MathXL 

(N=60) 

25 

8 

Reflex (N=97) 

34 

24 

ST Math 
(N=830) 

70 

23 

Think 

Through Math 
(N=236) 

75 

36 

Total 

(N=2,933) 

56 

28 


Selected Practice for 


materials 

for 

homework 

developing 

skill 

fluency 

Review 

and 

re-teaching 

16 

6 

10 

0 

0 

0 

0 

0 

0 

0 

8 

0 

8 

4 

1 

53 

3 

0 

30 

29 

19 

13 

10 

5 

13 

15 

9 

15 

8 

7 


For the second survey question, the teachers were asked to describe their 
overall satisfaction with the mathematics software. The teacher responses 
were coded for the primary focus or topic area of their response. We discuss 
only the most common five types of responses, therefore, the percentages 
across categories may not sum to 100%. Overall, 57% of the teachers report¬ 
ed overall general satisfaction (see Table 9). Many (11%) were satisfied with 
the way the product individualized instruction for the students. Ten percent 
reported being satisfied with student engagement while using the product. 

The third survey question asked teachers about their concerns with the 
mathematics technology. There were very few negative comments about the 
technology products. Six percent of teachers reported technical difficulties 
with the program (Table 10). Cognitive Tutor was the product with the high¬ 
est percent of negative responses where teachers reported student frustration. 
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However, it is important to note that the product and professional develop¬ 
ment package selected for this grant program was not the same package that 
Carnegie Learning usually provides for Cognitive Tutor implementation, 
but focused solely on the technology component for this implementation. 

Table 9 

Why Teachers Were Satisfied with Technology (Percent of Teachers) 


Categories 

Satisfied 

with 

provided 

technology 

Learning is 
adaptive and 
individualized 
for students 

Students 

are 

engaged 
when using 
technology 

Develops 
students’ 
knowledge 
or skills 

Student 

success 
or positive 
experience 

ALEKS 
(N=1,216) 

59 

16 

3 

3 

5 

Catchup Math 
(N=5) 

0 

20 

20 

0 

20 

Cognitive Tutor 
(N=15) 

40 

0 

0 

0 

0 

EdReady 

(N=12) 

0 

8 

0 

0 

17 

i-Ready 
(N=462) 

MathXL 

20 

7 

6 

1 

2 

(N=60) 

Reflex 

53 

8 

2 

0 

2 

(N=97) 

62 

6 

20 

20 

6 

ST Math 
(N=830) 

77 

5 

18 

9 

7 

Think Through 
Math (N=236) 

Total 

52 

19 

22 

17 

8 

(N=2,933) 

57 

11 

10 

6 

5 
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Table 10 

What Teachers Did Not Like about Technology (Percent of Teachers) 


Categories 

Product 

technical 

problems 

Not used 
the 

technology 

yet 

Student 
frustration 
or difficulty 

Lack of 
challenge 
or boring to 
students 

Need 
more time 
to use the 
product 

ALEKS 
(N=1,216) 

5 

9 

2 

2 

2 

Catchup Math 
(N=5) 

0 

0 

0 

0 

0 

Cognitive Tutor 
(N=15) 

7 

0 

13 

0 

7 

EdReady (N=12) 

0 

0 

0 

0 

0 

i-Ready (N=462) 

5 

2 

1 

2 

7 

MathXL 






(N=60) 

2 

12 

5 

0 

2 

Reflex 






(N=97) 

0 

3 

0 

2 

1 

ST Math 
(N=830) 

7 

2 

3 

2 

1 

Think Through 
Math (N=236) 

10 

1 

10 

6 

1 

Total 






(N=2,933) 

6 

5 

3 

2 

2 


The fourth survey question asked teachers to report challenges with in¬ 
tegrating these mathematics products into their class instruction. Thirty-two 
percent of teachers reported lack of technology access to be the greatest bar¬ 
rier (Table 11). The next most common challenge was in setting up student 
accounts, with the largest percent of teachers with concerns reported for 
MathXL (12%) and ST Math (10%). 
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Table 11 

Challenges Reported for Technology Integration (Percent of Teachers) 


Categories 

No barriers 

Not 

enough 

computers 

Licenses, 
accounts, 
and setup 

Lack of 
home 
access 

No or 
little 

use 

ALEKS 
(N=1,216) 

37 

31 

2 

4 

4 

Catchup Math 

(N=5) 

60 

40 

0 

0 

0 

Cognitive Tutor 
(N=15) 

20 

27 

0 

7 

0 

EdReady (N=12) 

33 

17 

8 

0 

0 

i-Ready (N=462) 

MathXL 

29 

29 

3 

2 

4 

(N=60) 

Reflex 

37 

22 

12 

7 

0 

(N=97) 

48 

25 

0 

8 

3 

ST Math (N=830) 

30 

37 

10 

3 

3 

Think Through 
Math (N=236) 

Total 

32 

30 

6 

7 

4 

(N=2,933) 

34 

32 

5 

4 

4 


The fifth survey question asked teachers how they used the performance 
management features of these mathematics products. The most common re¬ 
sponse was teachers reporting use of these features to monitor student prog¬ 
ress (34%). Approximately 20% of teachers surveyed who used Catchup 
Math and i-Ready used these features for generating reports for student’s 
Individualized Education Plans or to guide instruction for Response to In¬ 
tervention (Table 12). 

The data collection and reporting features are one of the primary 
strengths of these software programs; therefore, it was unfortunate to see 
so few teachers using these features to guide instruction and support for 
students (Table 12). In talking to teachers at a few participating schools we 
have learned that the first year they were focused more on logistics (e.g., ac¬ 
cess to computers) and this second year they have had time to plan for bet¬ 
ter integration and are beginning to find value in the data reports. 
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Table 12 

Use of Performance Management Features (Percent of Teachers) 


Categories 

Monitor 

students’ 

progress 

Did 

not 

Use 

Guide 

instruction 

Used to 
determine 
product 
usage 

Used for 
student 
lEPorRTI 

ALEKS 

(N=1216) 

31 

21 

15 

17 

4 

Catchup Math 
(N=5) 

20 

0 

0 

40 

20 

Cognitive Tutor 
(N=15) 

0 

27 

0 

20 

0 

EdReady 

(N=12) 

50 

8 

8 

8 

0 

i-Ready 
(N=462) 

MathXL 

29 

1 

9 

1 

19 

(N=60) 

Reflex 

35 

8 

12 

12 

10 

(N=97) 

57 

3 

4 

9 

5 

ST Math 
(N=830) 

31 

38 

11 

4 

9 

Think Through 
Math (N=236) 

Total 

61 

4 

9 

6 

12 

(N=2,933) 

34 

20 

12 

10 

9 


LIMITATIONS 

Prior research has shown many factors to be important for successful 
technology integration, such as the type of technology, the usage inten¬ 
sity, and different teacher factors. Studying the mediation effects of these 
factors was outside the scope of this study, since the purpose of this study 
was for accountability for the funding to determine overall impact on stu¬ 
dent achievement by product to inform future purchasing decisions. Future 
research can consider whether there are different types of technology with 
different affordances associated with differential effects on student achieve¬ 
ment or if there is an interaction between teacher factors and usage and re¬ 
lated effects on achievement. 
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A potential confound in the research design is that data was not collected 
on the types of mathematics technology software being used by students 
in the state in the comparison group. It may be that some were using very 
similar types of mathematics technology products, which may have reduced 
the ability to detect a significant effect of supplemental technology use on 
achievement. 

Another limitation is that our logistic regression was conducted using 
original data, and did not screen for common support (Caliendo & Ko- 
peinig, 2008) between treatment and comparison groups. We learned of this 
approach after the results had been presented to the stakeholders. We will 
consider this approach in year 2 when we will conduct a similar set of anal¬ 
yses with data from the second year of implementation. 

Our survey data analysis was conduced over a short period of time (ap¬ 
proximately two weeks) where eight graduate research assistants were in¬ 
volved in coding the open-ended teacher responses. Due to time constraints 
of needing to get the report to the stakeholders funding the project, we re¬ 
stricted our final analysis and summary of the data to the most common 
coding categories. If this area of research were our primary interest, we 
would have had multiple coders for each product data and have analyzed 
reliability among the coders. Therefore, the information we provide is for 
the purpose of context to note similarities in teacher feedback overall and 
by product. 


CONCLUSIONS 

We believe that these results from the first year of implementation of 
mathematics educational technology show the promise of these types of pro¬ 
grams in providing individualized instruction, practice, and automatic feed¬ 
back to students. Schools continue to be faced with pressures of accountabili¬ 
ty and the desire for all students to succeed. It is difficult to guarantee that all 
students will receive high quality instruction in mathematics each day. How¬ 
ever, with technology students and their parents have access to mathematics 
instruction that can provide remediation and acceleration. Rather than turning 
in an assignment and having to wait several days for feedback, which may be 
just a mark noting correct/incorrect responses, students now can have access 
to prescriptive feedback highlighting the errors made with some products 
even providing a re-teaching experience. 

Our findings are consistent with prior research, which has shown that 
teachers have challenges integrating technology into the classroom. Prior re¬ 
search has shown that infrastructure (Armstrong, 2014) can be a challenge to 
technology integration, which was confirmed by 32% of teachers responding 
to our survey who reported that they lacked sufficient access to technology. 
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While only 34% reported use of the performance management features of 
the products, it is not clear whether this relates to teachers’ technical skills 
as reported by prior research to be a challenge (Ertmer, 1999) or whether it 
was not as high of a priority as logistical issues of setting up accounts, gain¬ 
ing access to computers, and learning the basics of the software in this first 
year. 

While we find preliminary evidence to support impact of the technology 
on student achievement proficiency, this was in general when students were 
meeting the recommended level of usage. Of concern is that 90% of the stu¬ 
dents in the full sample were not meeting recommended usage. This may be 
due to the process of how this grant program ran through the district office, 
which may have slowed down distribution of licenses. This second year 
(2015-16) grants were limited to schools with usage during year 1, and the 
grant application required a letter of commitment from the school principal 
for access to technology for a minimum of 45 minutes per week for students 
using the educational technology. Mid-year the second year, close to 50% of 
students were meeting the recommended level of usage. 

IMPLICATIONS FOR FUTURE RESEARCH AND PRACTICE 

During this first year of implementation, teachers shared several differ¬ 
ent common models for integrating technology into the classroom. A model 
that seemed more common in elementary schools was the use of a technol¬ 
ogy station of four to six students with chrome books or computers for stu¬ 
dent use. This approach was reported as a way to differentiate instruction 
where the teacher could work with a small group, while stations were set 
up for students to rotate through independent work, with one station having 
the mathematics education technology. More common for middle school or 
junior high teachers was the use of computers on wheels (a rack of 25-30 
computers) shared between teachers in a hall or section of the building or 
the use of a computer lab. Future research should consider differences in 
implementation by grade group. 

The grant program restricted selection of products to one product per 
school with no child having access to more than one product. However, sev¬ 
eral schools have reported a concern that students may become bored using 
the same product for multiple years and have requested that future grants 
consider the option of using one product for grades K-3, another for grades 
4-6, etc. Product developers can also consider options for development that 
keep a student engaged, such as allowing the student to select a different 
avatar or setting, since there is some prior research evidence to support the 
impact of allowing students to personalize their learning experience (Walk- 
ington, 2013). Future research can consider whether impact decreases over 
time with student fatigue or boredom with a particular platform. 
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